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This presentation focuses on the application of computer graphics or “virtual reality” 
techniques as a human-computer interface tool in the operation of telerobotic systems. VR 
techniques offer very valuable task visualization aids for planning, previewing and predicting robotic 
actions, operator training, and for visual perception of non-visible events like contact forces in 
robotic tasks. The utility of computer graphics in telerobotic operation can be significantly enhanced 
by high-fidelity calibration of virtual reality images to actual TV camera images. This calibration 
will even permit the creation of artificial (synthetic) views of task scenes for which no TV camera 
views are available. 


OVERVIEW 

• VR as real-time control technique; fusion/ 
calibration issues 

• VR as training method; kinesthetic/tactile 
f eatu res/i nte rf aces 

• VR as task logistician; protocols and 
audio/vocal interfaces 

• VR as visualization tool for abstract, non- 
visible things; displays 

• Conclusion 
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VR AS REAL-TIME CONTROL TECHNIQUE WHEN FUSED 
WITH TV CAMERA IMAGES 


Task visualization is a key problem in teleoperation since most of the operator’s control 
decisions are based on visual information. The capability of previewing motions enhances the quality 
of teleoperation by reducing trial-and-error approaches in the hardware control and by increasing the 
operator’s confidence in control decision making during task execution. Predicting the consequences 
of motion commands under communication time delay permits the command of longer and safer 
action segments as opposed to the command of short action segments adopted in the move-and-wait 
control strategy in time-delayed teleoperation without predictive displays. 

Fusion of graphics and TV camera images can be generated by overlaying graphics images 
over actual TV camera images. A high-fidelity overlay requires a high-fidelity TV camera 
calibration and object localization. For this purpose, a reliable operator-interactive camera 
calibration and object localization technique has been developed at JPL during the past few years. It 
currently uses a point-to-point mapping procedure, and the computation of the camera calibration 
parameters is based on the ideal pinhole model of image formation by the camera. The technique 
uses the robot arm as the calibration fixture and assumes the use of a few selectable, good static TV 
camera views. The technique was demonstrated to a broad audience in May 1993 when a JPL control 
station was connected to an ORU exchange mock-up 4000 km away at GSFC through the NASA- 
select TV channel and the Internet computer network. The task was successfully performed under 
varying communication time delay conditions. 

The calibration technique and its application potential gained technical acceptance and is now 
being commercialized through a technical transfer agreement with DENEB Robotics, Inc. More on 
this calibration technique and its demonstration in Refs. 1 and 2. A narrated VCR tape is available on 
the demonstration in the JPL Audio-Visual Library (no. AVC-93-165C1D). 


• Motivation 

- Communication time delay: predict actions 

- Planning complex tasks: preview actions 

- Intelligent automation: supervise/monitor 
actions 

• Method of fusing VR images with TV images: 
create high-fidelity overlays of graphics images 
over TV camera images 

- Calibration technique: point or feature mapping 

- Motion control of graphics overlays 

• Extra benefit: enables artificial or synthetic views 
or scenes 

(See VCR tape for performance results) 
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VR AS TRAINING METHOD WITH KINESTHETIC/TACTILE 
FEATURES/INTERFACES 


Operator training using a VR display system is a convenient tool for initial familiarization of 
the operator with the teleoperated system without actually turning the hardware system on. Using 
proper physical modeling, even sensors and sensor fusion can be simulated and graphically shown to 
the operator. Computer graphics simulation of proximity sensor signals is a relatively simple task 
since it only implies the computation of distance from a fixed point of a moving robot hand in a given 
(computed) direction to the nearest environment surface in the graphics “world model”. Force-torque 
sensor signals can also be simulated by computing virtual contact forces and torques for given 
geometric contact models using a spring or a spring plus damper description of the actual contact 
interaction. For some detail, see Ref. 3. 

Modeling of soft things, like tissues, and graphically showing their deformations as a function 
of pressure is a very demanding undertaking. Some useful information on this topic can be found in 
Ref. 4. 


• Modeling of contact forces/moments and 
tactile area pressure - and showing them 
graphically 

• Modeling of “soft” things, like tissues - and 
showing their deformations graphically as 
a function of pressure 

• Application potential of manual force/ 
moment and tactile feedback from VR 
interaction scenes is increasing for 

- training operators (and surgeons) 

- “sensitive” teleprogramming 
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VR AS VISUALIZATION TOOL FOR DISPLAYING 
ABSTRACT, NON- VISIBLE THINGS 


Visualization of non-visible events enables a graphical representation of different non-visual 
sensor data and helps management of complex systems by providing a suitable graphical description 
of a multi-dimensional system state. For instance, the constrained and orientation restricted motion 
space of a dual-arm robot working in a closed kinematic chain configuration can be visualized as a 
complex 3-D object with hidden unreachable holes or cavities of varying shapes. An automated 
visualization method has been developed to find and visually represent this complex geometric object 
from a computed numerical data base. The method is an inverse computer vision technique in the 
sense that it creates rather than recognizes visual forms. More on this can be found in Ref. 5. 


• Several abstract, non-visible things can be 
present during telerobotic operations 
originating from 

- Internal system constraints - which can 
translate to complex task space constraints 

- Different multidimensional sensor data 
spaces - which, by some mapping, contain 
task/subtask goals as single event points or 
restricted small volumes 

• Visualization of non-visible things may require 
the use of 

- Complex computational procedures 

- Some artistic creativity for designing graphic 
forms 

• An example: visualization of a constrained 
dual-arm geometric work space treated as an 
inverse computer vision problem 
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VR AS TASK LOGISTICIAN SURROGATE 
WITH AUDIO/VOCAL INTERFACES 


In an emerging field of R&D, researchers embed task protocols or task scripts within a 
suitable VR representation. This requires to map a sequence of VR scenes to a sequence of required 
actions. In a related effort, researchers already initiated the idea of providing the operator with 
performance feedback messages on the operator interface graphics, derived from a stored model of 
the task execution protocol. A key element of such advanced feedback tool to the operator is a 
program that can follow the evolution of a teleoperated task by segmenting the sensory data stream 
into appropriate task performance phases. Task segmentation programs have already been 
implemented using Hidden Markov Model representations (Ref. 6) and Neural Network Architecture 
(Ref. 7) with very promising results. 


• VR with embedded task protocols/scripts - 
emerging field of R&D work 

- Mapping the sequence of VR scenes to 
sequence of required actions 

- Mapping the actual performance shown 
on VR scenes to required performance 

• Audio/vocal interface to VR-embedded 
task logistician is possible and desirable, 
with some operational restrictions 

- Fixed content both ways 

- Short statements both ways 

• Consideration of human factors 
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CONCLUSION 


The application of Virtual Reality techniques/tools in the operation of telerobotic 
systems enables the performance of more tasks, safer , faster , and inherently cheaper. 
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